{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 8: LangChain 集成\n",
    "\n",
    "[LangChain](https://python.langchain.com/docs/get_started/introduction.html)是一个用于开发由语言模型驱动的应用程序的流行库。您可以使用 LangChain 和 LLM 构建各种有趣的应用程序，例如 [聊天机器人](https://github.com/intel-analytics/BigDL/blob/main/python/llm/example/langchain/transformers_int4/chat.py)、[文档问答](https://github.com/intel-analytics/BigDL/blob/main/python/llm/example/langchain/transformers_int4/docqa.py)、[语音助手](https://github.com/intel-analytics/BigDL/blob/main/python/llm/example/langchain/transformers_int4/voiceassistant.py)。BigDL-LLM 提供 LangChain 集成（即 LLM wrapper 和 embedding），您可以像使用 [LangChain 中的其他 LLM wrapper](https://python.langchain.com/docs/integrations/llms/) 一样使用它们。\n",
    "\n",
    "本 Notebook 将介绍如何使用 langchain 与 BigDL-LLM 交互。\n",
    "\n",
    "## 8.1 安装"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，在准备好的环境中安装 BigDL-LLM。有关环境配置的最佳实践，请参阅本教程的 [第二章](../ch_2_Environment_Setup/README.md)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install bigdl-llm[all]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来安装 LangChain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U langchain==0.0.248"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **注意**\n",
    "> \n",
    "> 我们建议使用 `langchain==0.0.248`，这个版本在我们的教程中不会出现问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.2 LLM Wrapper\n",
    "\n",
    "BigDL-LLM 提供了 `TransformersLLM` 和 `TransformersPipelineLLM` ，它们实现了 LangChain 的 LLM wrapper 的标准接口。\n",
    "\n",
    "您可以使用 `TransformerLLM.from_model_id` 从 huggingface model_id 或路径实例化 `TransformerLLM`。您可以在 `model_kwargs` 中以字典形式传入与模型生成相关的参数（如 `temperature`, `max_length`）。让我们以 [`vicuna-7b-v1.5`](https://huggingface.co/lmsys/vicuna-7b-v1.5) 模型为例实例化创建一个 `TransformerLLM`。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bigdl.llm.langchain.llms import TransformersLLM\n",
    "\n",
    "llm = TransformersLLM.from_model_id(\n",
    "        model_id=\"lmsys/vicuna-7b-v1.5\",\n",
    "        model_kwargs={\"temperature\": 0, \"max_length\": 1024, \"trust_remote_code\": True},\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **注意**\n",
    ">\n",
    "> `TransformersPipelineLLM` 的实例化方式与 `TransformersLLM` 类似，都是通过 huggingface model_id 或路径、`model_kwargs` 和 `pipeline_kwargs` 来实现的。此外，还有一个额外的 `task` 参数用于指定要执行的任务类型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 prompt 模板来格式化 prompt，然后调用 `llm` 测试生成的结果。\n",
    "\n",
    "> **注意**\n",
    ">\n",
    "> `max_new_tokens` 参数定义了生成的 token 的最大数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AI stands for \"Artificial Intelligence.\" It refers to the development of computer systems that can perform tasks that typically require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. AI can be achieved through a combination of techniques such as machine learning, natural language processing, computer vision, and robotics. The ultimate goal of AI research is to create machines that can think and learn like humans, and can even exceed human capabilities in certain areas.\n"
     ]
    }
   ],
   "source": [
    "prompt = \"What is AI?\"\n",
    "VICUNA_PROMPT_TEMPLATE = \"USER: {prompt}\\nASSISTANT:\"\n",
    "result = llm(prompt=VICUNA_PROMPT_TEMPLATE.format(prompt=prompt), max_new_tokens=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "您也可以在 LLM 上使用 `generate` 来获取多组结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_result = llm.generate([VICUNA_PROMPT_TEMPLATE.format(prompt=\"Tell me a joke\"), VICUNA_PROMPT_TEMPLATE.format(prompt=\"Tell me a poem\")]*3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------number of generations--------------------\n",
      "6\n",
      "--------------------the first generation--------------------\n",
      "USER: Tell me a joke\n",
      "ASSISTANT: Why did the tomato turn red?\n",
      "\n",
      "Because it saw the salad dressing!\n"
     ]
    }
   ],
   "source": [
    "print(\"-\"*20+\"number of generations\"+\"-\"*20)\n",
    "print(len(llm_result.generations))\n",
    "print(\"-\"*20+\"the first generation\"+\"-\"*20)\n",
    "print(llm_result.generations[0][0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.3 使用 Chains"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，让我们开始使用 [Chains](https://docs.langchain.com/docs/components/chains/) 中的 LLM wrapper 和 embedding 。\n",
    "\n",
    ">**注意**\n",
    ">\n",
    "> Chain 是 LangChain 中的一个重要组件，它将一系列模块化组件（甚至是其他 chain）组合在一起，以实现特定目的。chain 中的组件可以是 prompt 模板、模型、内存缓冲区等。\n",
    "\n",
    "### 8.3.1 LLMChain\n",
    "\n",
    "首先，我们尝试使用一个简单的 chain `LLMChain`. \n",
    "\n",
    "如下创建一个简单的 prompt 模板："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import PromptTemplate\n",
    "\n",
    "template =\"USER: {question}\\nASSISTANT:\"\n",
    "prompt = PromptTemplate(template=template, input_variables=[\"question\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，使用我们在上一节创建的 `llm` 和刚刚创建的 prompt 模板来实例化一个 `LLMChain` 。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import LLMChain\n",
    "\n",
    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，让我们向 llm 问一个问题，并在 `LLMChain` 上调用 `run` 来获取响应。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AI stands for \"Artificial Intelligence.\" It refers to the development of computer systems that can perform tasks that typically require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. AI can be achieved through a combination of techniques such as machine learning, natural language processing, computer vision, and robotics. The ultimate goal of AI research is to create machines that can think and learn like humans, and can even exceed human capabilities in certain areas.\n"
     ]
    }
   ],
   "source": [
    "question = \"What is AI?\"\n",
    "result = llm_chain.run(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.3.2 Conversation Chain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在构建聊天应用程序时，我们可以使用更复杂的 chain 和内存缓冲区来记忆聊天记录。这对于实现多轮聊天体验非常有用。\n",
    "\n",
    "> **注意**\n",
    ">\n",
    "> `ConversationBufferMemory` 是 LangChain 中的一种存储器，用于存储对话中的信息并以不同格式提取出来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import PromptTemplate\n",
    "from langchain.chains import ConversationChain\n",
    "from langchain.chains.conversation.memory import ConversationBufferMemory\n",
    "\n",
    "template = \"The following is a friendly conversation between a human and an AI.\\\n",
    "    \\nCurrent conversation:\\n{history}\\nHuman: {input}\\nAI Asistant:\"\n",
    "prompt = PromptTemplate(template=template, input_variables=[\"history\", \"input\"])\n",
    "conversation_chain = ConversationChain(\n",
    "    verbose=True,\n",
    "    prompt=prompt,\n",
    "    llm=llm,\n",
    "    memory=ConversationBufferMemory(),\n",
    "    llm_kwargs={\"max_new_tokens\": 256},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI.    \n",
      "Current conversation:\n",
      "\n",
      "Human: Good morning AI!\n",
      "AI Asistant:\u001b[0m\n",
      "Good morning! How can I assist you today?\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "query =\"Good morning AI!\" \n",
    "result = conversation_chain.run(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI.    \n",
      "Current conversation:\n",
      "Human: Good morning AI!\n",
      "AI: The following is a friendly conversation between a human and an AI.    \n",
      "Current conversation:\n",
      "\n",
      "Human: Good morning AI!\n",
      "AI Asistant: Good morning! How can I assist you today?\n",
      "Human: Tell me about Intel.\n",
      "AI Asistant:\u001b[0m\n",
      "Intel is a multinational technology company that specializes in the development and manufacturing of computer hardware and technology solutions. It was founded in 1976 by Robert Noyce and Gordon Moore, and is headquartered in Santa Clara, California. Intel is best known for its microprocessors, which are the \"brains\" of most computers and devices. They also produce other hardware such as motherboard chipsets, flash memory, and other components. Intel is one of the largest and most well-known technology companies in the world, and is a leader in the development of new technologies and innovations in the field of computing.\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "query =\"Tell me about Intel.\" \n",
    "result = conversation_chain.run(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.3.3 MathChain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们尝试通过使用 `MathChain` 来让 LLM 解决一些数学问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **注意** \n",
    "> MathChain 通常需要使用较大的 `max_length` 来实例化 LLM，例如 1024\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import LLMMathChain\n",
    "\n",
    "MATH_CHAIN_TEMPLATE =\"Question: {question}\\nAnswer:\"\n",
    "prompt = PromptTemplate(template=MATH_CHAIN_TEMPLATE, input_variables=[\"question\"])\n",
    "llm_math = LLMMathChain.from_llm(prompt=prompt, llm=llm, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
      "What is 13 raised to the 2 power"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13 raised to the 2 power is equal to 13 \\* 13, which is 169.\n",
      "\u001b[32;1m\u001b[1;3mQuestion: What is 13 raised to the 2 power\n",
      "Answer: 13 raised to the 2 power is equal to 13 \\* 13, which is 169.\u001b[0m\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Answer:  13 raised to the 2 power is equal to 13 \\\\* 13, which is 169.'"
      ]
     },
     "execution_count": 256,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"What is 13 raised to the 2 power\"\n",
    "llm_math.run(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.4 通过文档回答问题\n",
    "\n",
    "如果您有一些文本文档（PDF、博客、Notion 页面等），并想要询问与这些文档内容相关的问题。鉴于 LLM 在理解文本方面的熟练程度，它是做这方面工作的一个很好用的工具。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.4.1 安装\n",
    "\n",
    "请在运行示例前安装必要的依赖库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U faiss-cpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.4.2 加载文档\n",
    "\n",
    "为方便起见，这里我们使用文本字符串作为加载文档。非结构化的数据可以从许多来源加载。请访问 [LangChain integration hub](https://integrations.langchain.com/) 来浏览全部加载器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_doc = \"\\\n",
    "    BigDL: fast, distributed, secure AI for Big Data\\n\\n\\\n",
    "    BigDL seamlessly scales your data analytics & AI applications from laptop to cloud, with the following libraries:\\\n",
    "        Orca: Distributed Big Data & AI (TF & PyTorch) Pipeline on Spark and Ray\\\n",
    "        Nano: Transparent Acceleration of Tensorflow & PyTorch Programs on XPU\\\n",
    "        DLlib: “Equivalent of Spark MLlib” for Deep Learning\\\n",
    "        Chronos: Scalable Time Series Analysis using AutoML\\\n",
    "        Friesian: End-to-End Recommendation Systems\\\n",
    "        PPML: Secure Big Data and AI (with SGX Hardware Security)\\\n",
    "        LLM: A library for running large language models with very low latency using low-precision techniques on Intel platforms\\n\\n\\\n",
    "    \""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.4.3 拆分输入文件的文本\n",
    "\n",
    "[Text splitters](https://python.langchain.com/docs/modules/data_connection/document_transformers/) 将文档分割成指定大小的片段。在这里，我们将文档分割成块，用于嵌入和向量存储。\n",
    "\n",
    "> **注意**\n",
    "> \n",
    "> `CharacterTextSplitter` 只对分隔符（默认为 `'\\n\\n'`）进行分割。\n",
    ">\n",
    "> `chunk_size` 是最大的分割字符数，前提是如果可以分割的话。\n",
    ">\n",
    "> `chunk_overlap` 是每次分割之间的重叠字符数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "\n",
    "text_splitter = CharacterTextSplitter(chunk_size=650, chunk_overlap=0)\n",
    "texts = text_splitter.split_text(input_doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.4.4 创建嵌入并存储到向量存储中\n",
    "\n",
    "拆分文档后，我们需要存储拆分内容，以便日后根据输入查询进行搜索。最常见的方法是嵌入每个分片的内容，然后将嵌入向量存储在向量存储中。\n",
    "\n",
    "众所周知，在 Transformers 中，有一些嵌入层可以将非结构化数据转换为嵌入向量（一个由数字组成的数组），从而对其执行各种操作。嵌入向量代表现实世界中的对象和概念，如单词、文档等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BigDL-LLM 提供了 `TransformersEmbeddings`，它允许你使用 LLM 从文本输入中获取嵌入。\n",
    "\n",
    "TransformersEmbeddings \"的实例化方法与 \"TransformersLLM \"类似"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bigdl.llm.langchain.embeddings import TransformersEmbeddings\n",
    "\n",
    "embeddings = TransformersEmbeddings.from_model_id(model_id=\"lmsys/vicuna-7b-v1.5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "介绍完 `TransformersEmbeddings` 后，让我们来创建嵌入并存储到向量存储中。向量存储负责存储嵌入数据并执行向量搜索。这里我们以 [Faiss](https://faiss.ai/index.html)为例，Faiss 是一个用于对密集向量进行高效相似性搜索和聚类的库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.vectorstores import FAISS\n",
    "\n",
    "docsearch = FAISS.from_texts(texts, embeddings, metadatas=[{\"source\": str(i)} for i in range(len(texts))]).as_retriever()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.4.5 获取相关文档\n",
    "\n",
    "如前所述，嵌入向量可以用于表示查询和文档。这种表示法可以将语义相似性转化为向量空间中的接近性。因此，我们可以通过这种相似性来搜索文档。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------number of relevant documents--------------------\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "query = \"What is BigDL?\"\n",
    "docs = docsearch.get_relevant_documents(query)\n",
    "print(\"-\"*20+\"number of relevant documents\"+\"-\"*20)\n",
    "print(len(docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.4.6 准备 chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains.chat_vector_db.prompts import QA_PROMPT\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "\n",
    "doc_chain = load_qa_chain(\n",
    "    llm, chain_type=\"stuff\", prompt=QA_PROMPT\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8.4.7 生成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BigDL is a fast, distributed, and secure AI library for Big Data. It enables seamless scaling of data analytics and AI applications from laptops to the cloud. BigDL supports various libraries, including Orca, Nano, DLlib, Chronos, Friesian, PPML, and LLM. These libraries cater to different use cases, such as distributed Big Data processing, transparent acceleration of TensorFlow and PyTorch programs, scalable time series analysis, end-to-end recommendation systems, and secure Big Data and AI with SGX hardware security. BigDL aims to provide a unified platform for AI and data analytics, making it easier for developers to build and deploy their applications at scale.\n"
     ]
    }
   ],
   "source": [
    "result = doc_chain.run(input_documents=docs, question=query)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm-langchain",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
